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Abstract 


The  Joint  Advanced  Distributed  Simulation  Joint  Test  Force  (JADS  JTF)  is  chartered  by  the  U.S.  Office  of  the 
Secretary  of  Defense  (OSD)  to  determine  the  utility  of  advanced  distributed  simulation  (ADS)  for  both 
developmental  and  operational  test  and  evaluation  (DT&E  and  OT&E).  The  program  is  nearing  its  completion 
and  this  paper  is  designed  to  provide  a  report  on  the  utility  of  ADS  in  enabling  the  distributed  test  and  evaluation 
(T&E)  of  aerospace  systems. 

The  paper  opens  with  a  brief  overview  of  ADS  technology  and  distributed  T&:E,  then  provides  a  short  description 
of  the  JADS  Joint  Test  and  Evaluation  (JT&E)  program.  The  main  portion  of  the  paper  will  discuss  results, 
lessons  learned  and  conclusions  derived  by  the  JADS  JTF. 

As  this  paper  uses  material  directly  from  JADS  reports  prepared  by  many  members  of  the  JADS  JTF,  the  credit  for 
this  paper  goes  to  them.  Readers  are  encouraged  to  seek  more  information  on  the  JADS  web  site 
(http://www.jads.abq.com)  which  will  be  active  until  March  2001. 

Overview  of  ADS 


Since  the  mid-1980s,  rapidly  evolving  information  systems  technology  has  been  put  to  work  in  support  of  U.S. 
Department  of  Defense  (DoD)  needs.  Early  efforts  were  conducted  jointly  by  the  Defense  Advanced  Research 
Projects  Agency  and  the  U.S.  Army.  This  early  project  was  named  Simulation  Network  (SIMNET)  and  it  was 
sharply  focused  on  training  applications.  Conceptually,  the  project  was  directed  toward  linking  training  devices 
(simulators)  with  human  operators  in  the  loop  at  distributed  sites  in  a  common  virtual  environment  in  near  real 
time.  SIMNET  evolved  to  distributed  interactive  simulation  (DIS),  a  technology  implementation  which  is  more 
flexible  and  far  reaching.  Formal  industry  standards  have  been  established  for  DIS.  In  turn,  DIS  is  giving  way  to 
high  level  architecture  (HLA),  a  technical  approach  championed  by  the  U.S.  Defense  Modeling  and  Simulation 
Office. 


JADS  uses  a  more  generic  term  for  the  technology  -  ADS.  This  is  defined  as  the  technology  and  procedures  that 
provide  a  time  and  space  coherent,  interactive  synthetic  environment  through  geographically  distributed  and 
potentially  dissimilar  simulations.  Any  combination  of  live,  virtual,  or  constructive  simulation  of  people  and/or 
equipment  can  be  used.  ADS  is  the  concept;  DIS  and  HLA  are  applications  of  ADS. 

Overview  of  Distributed  T&E 


ADS  is  the  enabling  technology  that  allows  the  T&E  professional  or  developmental  program  office  to  link 
geographically  separate  resources  together  for  a  T&E  event.  Examples  of  assets  that  can  be  linked  together  include 
digital  models  of  your  system  under  test  (SUT),  digital  representations  of  weapon  systems  or  weapon  flyouts, 
wargaming  models,  hardware-in-the-loop  or  system  integration  laboratories,  human-in-the-loop  simulators,  and 
actual  pieces  of  equipment  either  in  a  laboratory  or  field  condition. 

Distributed  T&E  architectures  can  be  tailored  to  meet  the  need  of  the  user  in  two  ways.  First,  just  the  assets  needed 
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for  a  particular  test  may  be  linked,  avoiding  unnecessary  cost  and  complexity.  Second,  the  level  of  fidelity  needed 
interaction  may  be  linked.  For  instance,  if  you  need  the  inputs  from  a  live  unmanned  aerial  vehicle 
(UAV),  It  can  be  linked  in.  If  not,  perhaps  a  digital  representation  of  the  UAV  is  adequate. 

Disfributed  T&E  provides  the  user  the  opportunity  to  gain  more  resources  for  a  test  than  what  may  normally  be 
available,  as  well  as  introduce  higher  levels  of  fidelity  into  the  system  development  process  than  has  been 
traditionally  available. 


Overview  of  JAPS  JT&E 


Background 


Because  of  widespread  interest  in  using  ADS  technology  to  support  T&E,  the  JADS  JT&E  program  was  nominated 
or  feasibility  study  in  1993.  The  nomination  was  motivated  by  the  T&E  community's  concern  about  long-standing 
test  constraints  and  limitations,  and  the  potential  utility  of  ADS  for  relieving  some  of  those  constraints  and 
limitations.  However,  there  was  widespread  skepticism  that  ADS  might  not  be  able  to  deliver  high-quality  data 
demanded  by  the  T&E  community.  The  Services  concurred  with  the  need  for  a  rigorous  examination  of  ADS 

utility  to  testing  and  OSD’s  Director  of  Test,  System  Engineering  and  Evaluation  chartered  JADS  as  a  full  joint 
test  program. 

JADS  JT&E  Charter 


The  basic  JADS  JT&E  program  was  chartered  in  October  1994  to  investigate  the  utility  of  ADS  for  both  DT&E 
and  OT&E.  More  specifically,  JADS  was  to  investigate  the  present  utility  of  ADS,  to  identify  critical  constraints 
in  using  the  technology,  to  develop  the  methodologies  in  using  ADS  in  various  T&E  applications,  and  to  provide 
growth  requirements  for  ADS  so  that  as  it  matures  it  better  meets  the  needs  of  the  T&E  community. 

At  the  time  of  chartering,  OSD  tasked  JADS  to  investigate  the  possibility  of  specifically  examining  ADS  utility  to 
Electronic  Warfare  (EW)  T&E.  This  additional  facet  of  the  program  was  subsequently  chartered  in  August  1996. 

Test  Approach 

To  accomplish  this  charter,  JADS  conducted  three  series  of  ADS-enhanced  tests  in  widely  different  areas  to 
determine  the  utility  of  ADS.  Representative  “systems  under  test”  were  used,  ones  that  had  already  undergone 
testing  and  had  been  fielded.  Significant  system  performance  data  was  then  available  for  comparison  with  the  data 
obtained  in  the  tests  using  ADS  as  a  methodology.  The  three  specific  test  programs  were  the  System  Integration 
Test  (SIT)  utilizing  two  air-to-air  missiles  [AIM-9M  Sidewinder  and  AIM-120  Advanced  Medium  Range  Air-to- 
Air  Missile  (AMRAAM)];  the  End-to-End  Test  (ETE)  using  the  Joint  Surveillance  Target  Attack  Radar  System 
(Joint  STARS)  as  a  representative  command,  control,  communications,  computer,  intelligence,  surveillance  and 
reconnaissance  (C4ISR)  system;  and  the  EW  Test,  utilizing  the  ALQ-I31  self-protection  jammer  (SPJ). 

System  Integration  Test 

SIT  evaluated  the  utility  of  using  ADS  to  support  cost-effective  testing  of  an  integrated  missile  weapon/launch 
aircraft  system  in  an  operationally  realistic  scenario.  The  purpose  of  SIT  also  included  the  evaluation  of  the 
capability  of  the  JADS  Test  Control  and  Analysis  Center  (TCAC)  to  control  a  distributed  test  of  this  type  and  to 
remotely  monitor  and  analyze  test  results. 

SIT  consisted  of  two  phases,  each  of  which  culminated  in  fully  linked  missions.  The  missions  simulated  a  single 
shooter  aircraft  launching  an  air-to-air  missile  against  a  single  target  aircraft.  In  the  Linked  Simulators  Phase 
(LSP),  the  shooter,  target,  and  missile  were  all  represented  by  hardware-in-the-loop  (HWIL)  laboratories  LSP 
testing  was  completed  in  November  1996.  In  the  Live  Fly  Phase  (LFP),  the  shooter  and  target  were  represented  by 
live  aircraft  and  the  missile  by  a  HWIL  laboratory.  LFP  testing  was  completed  in  October  1997. 
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Linked  Simulators  Phase.  The  LSP  test  concept  was  to  replicate  a  previous  AIM-9M-8/9  live  fire  profile  in  an 
ADS  configuration  and  compare  missile  results  for  the  LSP  trials  to  those  from  the  live  fire  test.  The  LSP  test 
configuration  is  shown  in  Figure  1. 


Figure  1 .  Linked  Simulator  Phase  Configuration 

The  F/A-18  Weapon  System  Support  Facility  (WSSF)  at  China  Lake,  California,  and  the  F-14D  Weapon  System 
Integration  Center  (WSIC)  at  Point  Mugu,  California,  were  the  shooter  and  target,  respectively.  The  shooter 
“fired”  the  AIM-9  in  the  Simulation  Laboratory  (SIMLAB)  HWIL  facility  at  the  target  which  could  respond  with 
countermeasures.  Runs  were  controlled  from  a  test  control  center  which  ensured  all  nodes  were  ready  for  each  run, 
issued  start/stop  directions,  and  processed  data  packets  for  real-time  analysis  of  system  performance.  Test  control 
was  exercised  from  the  Battle  Management  Interoperability  Center  (BMIC)  at  Point  Mugu  while  the  JADS  JTF 
was  physically  relocating.  Control  switched  to  the  JADS  TCAC  in  Albuquerque,  New  Mexico  after  the  move  was 
complete. 

Live  Fly  Phase.  The  LFP  test  concept  was  to  replicate  previous  AMRAAM  live  fire  profiles  in  an  ADS 
configuration  and  compare  missile  results  from  the  LFP  trials  to  those  from  the  live  fire  tests.  In  the  LFP,  ADS 
techniques  were  used  to  link  two  live  F-16  aircraft  (flying  on  the  Gulf  Test  Range  at  Eglin  Air  Force  Base,  Florida) 
representing  the  shooter  and  target  to  an  AMRAAM  HWIL  laboratory  (also  at  Eglin)  representing  the  missile. 

This  configuration  allowed  data  from  live  sources  to  drive  the  HWIL  laboratory  for  more  realistic  missile  results 
and  is  shown  in  Figure  2. 
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LIVE  AIRCRAFT  MISSILE 


Figure  2.  Live  Fly  Phase  Configuration 

Global  positioning  system  (GPS)  and  telemetry  (TM)  data  were  downlinked  from  the  aircraft  and  passed  to  the 
Central  Control  Facility  (CCF)  at  Eglin.  GPS,  inertial  navigation  system  (INS),  and  tracking  radar  data  for  each 
aircraft  were  combined  by  the  time-space-position  information  (TSPI)  Data  Processor  (TDP)  in  the  CCF  to  produce 
optimal  entity  state  solutions.  The  aircraft  entity  state  data  were  transformed  into  DIS  protocol  data  units  (PDUs) 
and  transferred  to  the  AMRAAM  HWIL  simulation  at  the  Missile  Simulation  Laboratory  (MISILAB)  over  a  T-3 
link.  The  shooter  aircraft  “fired”  the  AMRAAM  in  the  MISILAB  at  the  target  and  provided  data  link  updates  of 
the  target  position  and  velocity  to  the  missile  during  its  flyout.  The  AMRAAM  seeker  was  mounted  on  a  flight 
table  and  responded  to  radio  frequency  (RF)  sources  in  the  MISILAB  which  simulated  the  seeker  return  tfom  the 
target,  the  relative  motions  of  the  target  and  the  missile,  and  electronic  countermeasures  (ECM).  A  link  between 
the  CCF  and  the  JADS  TCAC  allowed  JADS  personnel  to  monitor  and  record  the  simulated  intercepts. 

End-to-End  Test 


The  ETE  used  distributed  simulations  to  assemble  an  enhanced  environment  to  be  used  for  testing  C4ISR  systems. 
The  object  was  to  determine  if  ADS  can  provide  a  complete,  robust  set  of  interfaces  from  sensor  to  weapon  system 
including  the  additional  intermediate  nodes  that  would  be  found  in  a  tactical  engagement.  The  test  traced  a  thread 
of  the  battlefield  process  from  target  detection  to  target  assignment  and  engagement  at  corps  level  using  ADS. 
Figure  3  illustrates  the  basic  test  architecture. 


UNCLASSIFIED 


UNCLASSIFIED 


Figure  3.  End-to-End  Test  Architecture 


The  ETE  was  a  four-phased  test.  Phase  1  was  largely  developmental  —  constructing  the  various  components 
necessary  to  executing  later  phases  of  testing.  These  components  included  a  high  fidelity  emulation  of  the  Joint 
STARS  radar  processes,  called  Virtual  Surveillance  Target  Attack  Radar  System  (VSTARS),  which  included  both 
moving  target  indicator  and  synthetic  aperture  radar  modes  of  operation.  Phase  2  linked  representative  entities  for 
the  end-to-end  process  while  the  “system  under  test”  was  in  a  laboratory  environment  enabling  JADS  to  explore 
the  utility  of  ADS  in  the  DT&E  and  early  OT&E  of  a  C4I  system.  Phase  3  hosted  VSTARS  on  board  the  actual 
Joint  STARS  aircraft  and  performed  final  integration  testing.  Phase  4  was  an  actual  live  open  air  test  with  the 
aircraft  airborne,  with  the  environment  augmented  by  ADS. 

Electronic  Warfare  Test 


JADS  EW  Test  was  chartered  separately  by  OSD  to  examine  the  utility  of  ADS  in  EW  T&E.  To  allow  JADS  to 
conduct  a  broad  analysis  of  this  domain  and  remain  within  very  tight  fiscal  constraints,  a  “multi-vectored” 
approach  was  employed.  JADS  leveraged  off  the  U.S.  DoD  HLA  Engineering  Prototype  Federation  for  lessons 
learned  in  constructing  and  implementing  a  distributed  architecture  for  EW  T&E.  At  the  bequest  of  DoD’s 
CROSSBOW  Committee,  JADS  directed  the  Threat  Simulator  Linking  Activity  Study,  which  delineates  how  to 
link  DoD’s  EW  test  facilities  using  HLA.  Third,  JADS  participated  with  the  U.S.  Army  in  its  Advanced 
Distributed  Electronic  Warfare  System  test,  a  concept  that  provides  EW  effects  on  communications  gear  in  the 
open  air  environment  without  the  actual  EW  open  air  emissions. 

SPJ  Test 


The  SPJ  test  was  designed  as  a  three-phased  test  focusing  on  the  U.S.  DoD  EW  test  process,  and  utilized  the  ALQ- 
131  as  its  “system  under  test.”  Phase  1  was  a  non- ADS  test  of  the  SPJ  on  an  open  air  range  (OAR),  augmented 
with  data  obtained  by  testing  the  ALQ-13 1  in  an  HWIL  facility.  The  purpose  of  this  test  was  to  establish  a  baseline 
of  environment  and  performance  data  which  was  used  to  develop  the  ADS  test  environment  for  the  following 
phases  and  was  the  basis  for  determining  the  validity  of  ADS  test  results.  Phase  2  was  a  test  of  a  high-fidelity  real¬ 
time  digital  system  model  (DSM)  of  the  ALQ-131  linked  with  HWIL  terminal  threats  and  a  constructive  model  of 
an  Integrated  Air  Defense  System  (IADS).  The  threat  laydown  from  the  OAR  was  replicated  in  the  synthetic  ADS 
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environment  and  the  ALQ-131  was  flown,  via  a  scripted  flight  profile  developed  from  the  actual  OAR  flights, 
through  the  IADS,  engaging  the  high-fidelity  terminal  threats.  Phase  3  was  a  test  of  the  SPJ  installed  on  an  actual 
aircraft  located  in  an  Integrated  System  Test  Facility  (ISTF).  The  facility  was  linked  with  HWIL  threats  and  the 
constructive  model  of  the  IADS  using  the  same  threat  laydown  as  the  previous  test  and  controlled  by  the  same 
scripted  flight  profile.  Figure  4  illustrates  Phases  2  and  3  of  the  SPJ  test. 
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Figure  4.  Self-Protection  Jammer  Phases  2  and  3 


JAPS  JT&E  Test  Results,  Lessons  Learned  and  Conclusions 

This  section  will  briefly  summarize  the  test  results,  lessons  learned  and  conclusions  for  the  SIT  and  ETE  tests. 
The  phase  and  utility  reports  for  the  EW  test  are  being  prepared  at  the  time  of  this  writing,  so  are  not  included  in 
this  paper.  Because  of  the  fundamental  difference  between  the  SIT’s  two  test  phases,  LSP  and  LFP,  they  will  be 
addressed  separately. 


Linked  Simulators  Phase 


LSP  Test  Results  The  key  results  from  LSP  testing  are  as  follows: 

The  simulation  facilities  were  properly  linked,  and  the  missile  flyouts  were  valid  for  the  target 
representation  in  the  SIMLAB.  However,  this  target  representation  differed  somewhat  from  the  target 
data  originating  from  the  Weapon  System  Integration  Center  (WSIC). 

The  manual  method  for  replicating  a  given  profile  resulted  in  very  good  run-to-run  reproducibility  of  the 
engagements. 

The  average  latency  of  all  entity  state  data  during  the  final  mission  were  relatively  small  (<100 
milliseconds  from  simulation  to  simulation)  and  consistent  run-to-run.  However,  relatively  large  random 
latency  variations  were  often  observed  which  resulted  in  an  uncertainty  in  the  target  location,  as  perceived 
in  the  SIMLAB. 

The  ADS  network  provided  ample  bandwidth  and  no  loss  of  connectivity  during  testing. 

There  were  no  significant  ADS-induced  errors. 

The  reliability  of  the  long-haul  network  was  very  good,  and  the  availability  of  the  complete  LSP  ADS 
configuration  was  on  the  order  of  85%. 

Test  control  procedures  were  refined  throughout  the  preparation  process  and  worked  well  during  testing. 
LSP  Lessons  Learned  Key  LSP  technical  and  programmatic  lessons  learned  are: 
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Accurate  coordinate  transformations  are  necessary.  They  must  be  verified  and  validated  at  each  site  and 
then  revalidated  during  end-to-end  testing  as  early  as  possible  in  the  test  phase. 

Quantitative  validation  has  limitations.  JADS  intent  was  to  quantitatively  verify  missile  simulation 
performance  against  live  fire  data.  However,  as  only  one  live  fly  event  was  available  to  support  the 
process,  a  modified  approach  including  both  quantitative  and  qualitative  methods  was  used,  and 
successfully  identified  invalid  results. 

Network  interface  units  (NIUs)  need  improvement.  NIUs  are  necessary  if  two  nodes  cannot  communicate 
directly  in  a  common  language.  They  can  be  a  major  source  of  both  errors  and  processing  delays.  Better 
direct  user  control  of  the  content  of  the  data  and  network  communications  is  needed. 

Common  ADS-related  hardware  and  software  is  needed.  In  the  LSP,  it  was  difficult  to  get  the  ADS 
network  to  behave  in  a  uniform  fashion  due  to  the  many  different  types  of  interface  hardware, 
communications  equipment  (routers),  and  interface  software  versions. 

Latency  variations  were  significant.  Processing  delays  were  the  primary  culprit  here. 

Time  sources  must  be  synchronized  off  the  same  time  source  and  then  must  be  validated  at  each  test  site 
prior  to  project  operations  to  ensure  accurate,  synchronized  time  is  precisely  recorded  at  each  test  site. 
Special  test  equipment  is  needed  for  check-out  and  verification  of  the  ADS  architecture.  Without  this 
equipment,  trial  and  error  becomes  the  norm  when  (not  if)  problems  crop  up. 

The  requirements  for  an  ADS  test  must  be  clearly  defined  early  in  the  test  planning  phase.  This  includes 
user  requirements,  support  agency's  stated  actions,  and  operations  security  requirements.  Planning  and 
coordination  details  will  be  much  more  involved  than  in  a  traditional,  non- ADS  test. 

Get  “system  under  test”  experts  involved  from  the  beginning. 

Test  communications  requirements  must  be  addressed  early  in  the  test  planning  phase.  This  is  necessary 
to  ensure  effective  communications  during  the  test.  Also,  a  linked  test  should  have  multiple  (more  than 
two)  communications  nets  with  easy,  selectable  access  to  all  the  nets  from  multiple  locations  within  the 
site.  Finally,  the  capability  for  secure  video  teleconferencing  pays  big  dividends  during  planning, 
coordination,  and  post-test  debriefs. 

A  stepped  build-up  approach  should  be  used.  First,  a  systematic  check-out  of  the  stand-alone  simulators 
(live,  virtual  or  constructive)  is  needed.  Next,  direct  (non-DIS)  links  should  be  used  during  test  build-up. 
Finally,  structured  testing  of  the  network  must  be  performed  prior  to,  and  independent  of,  the  linked 
testing  times  and  the  simulation  laboratories  to  validate  transmission/reception  rates,  bandwidth 
utilization,  latency,  data  transmission  and  reception,  etc.,  prior  to  commencing  project  test  periods. 

Linking  of  facilities  using  ADS  can  require  significant  facility  interface  hardware  and  software 
development.  ADS  implementation  is  not  “plug  and  play,”  at  least  for  some  time. 

Local  (on-site)  test  monitoring/control  should  be  used  prior  to  remote  test  monitoring/control. 

Tight  control  of  the  aircrew  is  not  desirable.  Give  them  the  critical  parameters  and  switchology  to  meet 
the  test  objectives  and  allow  them  to  make  tactical  decisions,  fly  the  “aircraft,”  operate  the  weapon  system, 
etc. 

Additional  time  is  needed  before  the  beginning  and  after  the  end  of  each  testing  period.  One  hour  is 
recommended  for  set-up,  and  two  hours  at  the  end  for  data  logging,  data  archiving,  data  transfer,  and 
laboratory  reclassification. 

Briefings  are  needed  before  and  after  each  mission. 

Effective  data  management  is  needed,  as  ADS  can  generate  mountains  of  data.  A  comprehensive  plan 
will  clearly  identify  the  data  to  be  collected  at  each  site,  on-site  processing  of  the  data,  and  data  to  be 
transferred  to  the  analysis  center. 

Adequate  time  must  be  allowed  for  data  analysis  between  test  events.  Analysis  procedures  should  be 
rehearsed  to  better  understand  the  amount  of  time  needed  for  this  analysis. 

Configuration  control  is  essential.  This  one  obvious  area  was  one  of  great  challenge  considering  the  many 
sites  involved  and  the  multiple  uses  of  each  site. 

LSP  Conclusions  The  LSP  ADS  configuration  has  utility  for: 

Missile  weapon/launch  aircraft  system  integration  T&E. 

Parametric  studies,  due  to  good  pilot  manual  reproducability  of  the  profiles. 

Rehearsal  and  refinement  of  live  engagement  scenarios. 

T&E  with  closed-loop  shooter/target  interactions  (e.g.,  prelaunch  tactics  evaluation). 
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T&E  with  open-loop  missile/target  interactions  (e.g.,  the  missile  reacts  to  the  target,  but  the  target  does 
not  react  to  the  missile). 

Live  Fly  Phase 

LFP  Results  The  key  results  from  LFP  testing  are  as  follows: 

Live  aircraft  were  properly  linked  to  the  missile  HWIL  laboratory  and  the  MISILAB  generated  valid 
AMRAAM  data  during  the  engagement. 

Accurate  TSPI  solutions  were  generated  by  the  TSPI  Data  Processor  (TDP)  to  the  order  of  one  to  three 
meters  in  position  and  one  meter  per  second  in  velocity.  This  well  exceeded  MISILAB  accuracy 
requirements. 

The  shooter  and  target  TSPI  data  were  properly  synchronized  to  each  other  and  to  the  umbilical  and  data 
link  messages  for  input  to  the  MISILAB  simulation. 

Latencies  during  testing  were  relatively  stable  and  consistent,  but  fairly  large.  The  total  latency  of  the 
MISILAB  simulation  was  about  3.1  seconds.  This  large  value  of  latency  was  due  to  the  processing  and 
buffering  of  the  TSPI  data  to  produce  accurate  and  smooth  solutions  and  to  the  synchronization  technique 
used. 

The  ADS  network  provided  ample  bandwidth  and  no  loss  of  connectivity  during  testing. 

There  were  no  significant  ADS-induced  errors. 

Test  control  procedures  worked  well  during  testing  with  centralized  test  control  exercised  from  the  CCF. 

LFP  Lessons  Learned  Key  lessons  learned  from  LFP  are: 

As  in  the  LSP,  a  major  lesson  learned  is  that  stand-alone  simulation  facilities  (for  live,  virtual  or 
constructive  entities)  can  require  significant  modifications  before  effective  linking  is  possible. 

Additionally,  linking  may  require  special  purpose  interfaces  so  as  to  accept  inputs  in  real  time. 
Development  of  such  units  must  be  factored  into  test  planning. 

Key  interfaces  need  realistic  integration  testing.  Replaying  data  from  a  recorded  mission  worked  well  in 
most  cases  (and  was  most  cost  effective);  however,  some  integration  testing  required  a  live  mission. 

Early  definition  of  network  requirements  was  very  advantageous.  This  was  a  major  lesson  from  LSP  that 
JADS  took  advantage  of. 

Changes  and  upgrades  to  aircraft  instrumentation  delayed  development.  Specially  instrumented  aircraft 
were  required  to  support  the  LFP  flights.  Due  to  the  small  number  of  such  aircraft,  the  LFP  schedule  was 
very  sensitive  to  periodic  aircraft  phase  inspections,  software  upgrades,  and  higher  priority  missions. 
Merging  several  TSPI  sources  was  advantageous.  Real-time  aircraft  inertial  navigation  system  (INS)  and 
GPS  data  were  combined  to  calculate  more  accurate  kinematic  estimates.  When  combined  with  the 
ground  radars,  solutions  of  one  to  three  meters  in  position  and  one  meter  per  second  in  velocity  were 
achieved. 

A  strong  program  manager  or  system  integrator  is  needed  to  oversee  facility  development,  due  to  the 
difficulty  in  coordinating  several  diverse  facilities  to  successfully  integrate  an  ADS-linked  configuration. 
Use  risk  reduction  tests  for  integration.  A  building  block  approach  was  used  successfully  to  check  out 
interfaces  at  the  lowest  level,  then  one  or  two  resources  at  a  time  were  added  to  integrate  the  linked 
configuration.  These  risk  reduction  tests  were  also  useful  for  developing  analytical  tools. 

Several  subnetworks  should  be  used  for  voice  communications.  Three  voice  communications  networks 
were  needed  to  support  more  than  30  people  at  various  locations,  and  a  fourth  network  could  have  further 
aided  decision  making. 

Two-dimensional  displays  were  needed  at  each  node;  they  greatly  enhanced  the  situational  awareness  of 
the  'participants. 

Existing  range  procedures  had  to  be  modified  for  ADS.  The  existing  test  procedures  were  only  written  for 
individual  facilities,  so  a  new  combined  checklist  was  created  for  ADS  applications. 

Laboratory  replays  served  as  an  excellent  method  of  test  rehearsal. 

LFP  Conclusions  The  LFP  ADS  configuration  has  utility  for: 

Missile  weapon/launch  aircraft  system  integration  T&E,  especially  evaluation  of  the  targeting  messages 
supplied  to  the  missile  by  the  shooter. 
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Rehearsal  and  refinement  of  live  engagement  scenarios. 

Tactics  development  involving  closed-loop  interactions  between  the  shooter  and  target. 
Efficient  testing  utilizing  an  analyst-in-the-loop  for  timely  feedback  during  the  mission. 

End-to-End  Test 


ETE  Results  The  key  results  from  ETE  testing  were: 

Both  ADS  configurations  used  in  the  test  generated  valid  Joint  STARS  data,  as  judged  by  subject  matter 
experts. 

-  It  was  specifically  validated  that  Janus  6.88D  represented  vehicle  behavior  to  the  degree  detectable 
by  the  Joint  STARS,  as  judged  by  Joint  STARS  operators  viewing  vehicle  movement  presented  by  the 
Joint  STARS  operator  workstation. 

-  Test  participants  could  not  identify  any  Joint  STARS  radar  simulation  process  or  function  that 
limited  their  ability  to  perform  their  mission/job  or  altered  their  approach  to  their  mission/job. 

-  The  credibility  of  the  Joint  STARS  data  was  enhanced  by  the  use  of  actual  operational 
hardware  with  trained  operators  whenever  possible. 

Sufficient  data  were  collected  to  support  evaluation  of  most  of  the  Joint  STARS  multiservice  operational 
T&E  measures.  These  measures  were  not  evaluated  by  JADS  analysts  since  the  JADS  objective  was  to 
evaluate  ADS-based  testing,  not  Joint  STARS. 

When  live  and  virtual  entities  were  mixed  using  the  live  ETE  test  configuration,  it  was  impossible  to 
distinguish  between  them  based  upon  size,  color,  contrast,  and  movement.  However,  the  live  and  virtual 
areas  could  be  distinguished  based  on  the  different  scenarios  used  and  the  civilian  traffic  “clutter.”  The 
live  area  contained  civilian  traffic  around  Ft.  Hood,  and  the  virtual  area  contained  simulated  battlefield 
entities  in  Iraq.  If  the  live  entities  had  simulated  a  battle  in  Iraq,  the  live  and  virtual  entities  would  have 
been  indistinguishable. 

The  ETE  required  only  a  small  traction  of  the  T-1  bandwidth  (1 .544  megabits  per  second).  This  indicates 
that  a  much  higher  data  transfer  rate  (e.g.,  more  nodes  or  entities)  could  be  applied  to  distributed  testing 
without  causing  bandwidth  problems. 

The  live  testing  using  the  SATCOM  link  to  the  E-8C  aircraft  showed  that  all  the  available  SATCOM  link 
bandwidth  was  required  for  data  transmission,  and  that  buffering  was  needed  at  times  to  handle  periods  of 
heavy  scenario  activity.  Without  buffering,  the  SATCOM  link  exhibited  an  average  latency  of  around  two 
seconds.  With  buffering,  the  latency  approached  six  seconds. 

The  ETE  test  exhibited  acceptable  latency  values.  In  other  words,  the  latencies  experienced  during  testing 
did  not  affect  the  validity  of  test  results. 

The  ETE  test  network  was  highly  reliable  during  testing,  largely  because  of  the  ETE  test  team’s  extensive 
pretest  risk  reduction  efforts. 

Test  control  procedures  were  refined  throughout  the  preparation  process  and  worked  well  during  testing. 


ETE  Lessons  Learned  Key  lessons  learned  from  ETE  are: 

Simulations,  when  used  in  distributed  testing,  should  be  carefully  planned  and  developed.  Using  reliable 
simulations  is  important  for  a  successful  test. 

Distributed  testing  often  requires  linkage  among  dissimilar  facilities,  network  equipment,  and 
simulations.  However,  careful  planning  can  significantly  reduce  the  potential  for  difficulties  arising  from 
network  interface  problems. 

Time  sources  must  be  synchronized  using  a  “master  clock”  and  then  validated  at  each  network  node. 
Special  test  equipment  and  networking  tools  are  necessary  for  distributed  testing.  The  tool  set  must  be 
able  to  rapidly  isolate  the  specific  cause  of  network  and  ADS/DIS  problems. 

ADS  test  requirements  must  be  clearly  defined  early  in  the  test  planning  phase,  since  individual  facilities 
are  generally  unfamiliar  with  conducting  coordinated,  distributed  tests. 

Develop  contingency  plans  that  allow  quick  decisions  on  alternatives  and  prevent  unexpected  equipment 
and  communications  problems  from  completely  disrupting  the  test. 
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Use  a  stepped  approach  to  testing  where  each  successive  ADS  test  builds  on  the  success  of  earlier  tests. 
This  “test,  analyze,  fix,  test”  approach,  in  concert  with  structured,  independent  testing  of  the  network,  will 
greatly  improve  the  chances  for  successful  distributed  testing. 

Risk  reduction  testing  prior  to  actual  test  execution  provided  effective  rehearsals  and  was  helpful  for 
troubleshooting. 

Detailed  planning  for  data  management  is  necessary  before  testing  commences. 

Have  a  centralized  test  control  center  with  test  controllers  who  are  extremely  familiar  with  the  test  and 
network  configuration. 

-  Personnel  involved  in  a  distributed  test  need  to  understand  the  “big  picture.”  When  people  are 
geographically  separated,  it  becomes  easy  for  them  to  focus  on  their  own  individual  portion  of  the  test. 
When  problems  arise,  personnel  who  understand  the  entire  test  and  the  overall  network  will  find  solutions 
much  faster. 

ETE  Conclusions  Key  conclusions  from  the  ETE  Test  are: 

The  ETE  live  ADS  configuration  has  utility  for  realistic  OT&E  to  include  mission-level  testing  of  C4ISR 
systems. 

-  In  contrast  to  ADS-supported  testing,  most  traditional  live  C4ISR  testing  cannot  be  done  at  the 
mission  level  because  of  the  unavailability  and  prohibitive  cost  of  involving  the  required  numbers  of 
live  battlefield  entities  and  limited  access  to  key  real  world  systems  because  of  their  limited  numbers  and 
high  usage. 

-  The  use  of  ADS  permits  the  number  of  targets  to  be  greatly  augmented  by  the  seamless 

integration  of  live  and  virtual  threat  entities  while  operating  the  sensor/sensor  platform  in  the 
actual  operational  environment. 

-  Realism  and  credibility  of  test  results  are  enhanced  by  the  use  of  actual  operational  hardware  by 

trained  operators  and  by  the  employment  of  the  sensor  in  a  real  open-air  environment  subjecting  it 

to  actual  background  and  clutter  effects. 

The  ETE  laboratory  ADS  configuration  has  utility  for: 

-  C4ISR  DT&E,  since  laboratory/brass  board  SUT  configurations  can  be  used,  and  multiple 
repetitions  of  the  same  scenario  can  be  performed  for  parametric  testing. 

-  Realistic  rehearsal  and  refinement  of  live  test  scenarios,  since  multiple  repetitions  of  the  scenarios  can 
be  performed  using  the  same  equipment  operators  as  the  live  test. 

-  Early  operational  assessment  of  C4ISR  system  components  prior  to  building  them  by  using 

laboratory/brass  board  configurations  for  new  components  (e.g.,  sensor/sensor  platforms)  linked  to 
existing  tactical  hardware  manned  with  actual  operators  for  the  other  components  (e.g.,  command 

and  control  elements). 

Both  ADS  configurations  have  utility  for  C4ISR  system  training. 

—  Conventional  training  can  be  limited  by  the  availability  of  those  assets  making  up  the  C4ISR 

system’s  operational  environment.  ADS  technology,  by  simulating  those  key  assets,  can  provide  longer 
periods  of  time  for  realistic  operation  of  the  C4ISR  system. 

-  C4ISR  system  operators  can  take  advantage  of  the  additional  training  time  provided  by  ADS 

technology  to  confirm  current  tactics  and  to  test  “what  if’  scenarios  and  new  tactics. 

-  ADS  simulations  can  help  C4ISR  system  operators  familiarize  themselves  with  the  maneuver  tactics 
of  foreign  armed  forces  -  valuable  experience  for  possible  future  deployments. 

Electronic  Warfare  Test 


At  the  time  of  this  writing,  the  JADS  JTF  is  still  compiling  the  test  phase  and  EW  utility  reports.  All  of  these 
reports  will  be  available  on  the  JADS  web  site  (http://www.jads.abcj.com)  by  1  December  1999. 

Conclusions 


JADS  was  chartered  to  determine  the  truth  as  to  whether  ADS  is,  or  is  not,  a  feasible  tool  for  T&E.  Evidence  is 
that  ADS  can  bring  many  benefits  to  the  table.  However,  one  must  be  fully  aware  of  the  inherent  limitations  of  the 
technology.  Also,  one  would  be  well  advised  to  learn  from  those  who  have  practical  experience  in  using  ADS  in 
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the  T&E  arena.  Distributed  T&E  is  certainly  not  a  panacea  that  will  solve  all  of  the  problems  and  meet  all  of  the 
requirements.  However,  it  does  appear  to  be  a  powerful  tool,  if  used  appropriately  and  intelligently,  and  should  be 
considered  in  balance  with  other  methodologies  when  developing  a  T&E  program. 
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Acronyms 


Following  are  acronyms  found  in  this  paper’s  figures  and  not  defined  in  the  body  of  the  paper. 

Figure  1 
A/C  -  Aircraft 
IR  -  Infrared 
MSL  -  Missile 

SMS  ’  Stores  Management  System 
TGT  -  Target 

Figure  2 

AASI  ’  Aircraft  Avionics  Simulation  Interface 
ECM  -  Electronic  Countermeasures 
RDL  -  Rear  Data  Link 
UMB  -  Umbilical 
TM  -  Telemetry 

Figure  3 

ACE  -  Analysis  and  Control  Element 
ASAS  -  All-Source  Analysis  System 
ATACMS  -  Army  Tactical  Missile  System 
DOCC  -  Deep  Operation  Coordination  Center 
FDC  -  Fire  Direction  System 
FSE  -  Fire  Support  Element 
GSM  -  Ground  Station  Module 
LOS  -  Line  of  Sight 
OK  -  Oklahoma 

SATCOM  -  Satellite  Communications 
SCDL  -  Surveillance  Control  Data  Link 

TRAC  -  U.S.  Army  Training  and  Doctrine  Command  Analysis  Center 
WSMR  -  White  Sands  Missile  Range 

Figure  4 

ECM  -  Electronic  Countermeasures 
IADS  -  Integrated  Air  Defense  System 
RF  -  Radio  Frequency 
TECH  -  Technique 
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